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Objectives * , , « * ' • 

In the research proposal for this project, a variety of tasks 

were outlined, including: . K 

1. Algebraic formulation of a simplex grox^^th model. 

• 2. Formulation*ofi this model in the format r^quit^ 
J8reskog*s LISREL program. • 

^ ■ 

* 3. Testing the fit of this simplex mod^l to a Variety of 
longitudinal, academic mea^uces. 

4. ' 'Specify4.ng implicatioils 'for the study pf the determinants^ 
- of academic growth. . 

The* results for'tasks 1 and 2 dre given in Section C, thos^for: 

task 3 In sections C, D, and E, and those for task 4 mostly^ in 

section D. All of the original objectives were apcomplished. 
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Introduction j 

^ pince\ project of this type Is , of little value unless its 

. procedtires are^ made available to other researchers , the x^ork x^as 
■ ' ' . * 

organized so as to. produce publishable journal articles as soon as ; 
thjB relevant results became available,* It is these articles Xirhich • 
form the basis for this report i.e., 

1. Section C has been accepted by Educational and Psychological 
« Measurement x^ith the title, "A Simplex ^del for Analyzing 
\ Academic'' Grox^th** . ^ - - ' 

. ^ 2. Section F has been accepted by Educational and Psychological 
Measurement x^j.th the title "Analyzing Ratings with Correlated 
* Intrajudge Measurement Errors". » 

3. Section D^has been submitted to Eflucational and Psychological 
' ^ Measurement as ^'The Correlation of Status- With Gain". 

4.. Section E has been submitteid to Educational arfd Psycho logicajL 

«• • ■ 

• ■; • --ti ^ ■ ' ' 

Measurement as 'Vthe I^liability of College Grades from 

t * ' 

/ Longitudinal PatA". 

/ • • --r. ' ' • ' 

It Xijas cousidered desitable to submit all these ar tickles to tde same 

journal mostly because ! of the appropriateness of, Its readership but 

' \ ' ' ' , ' i ■ ' ■ ' ' * . 

alsb because these papers form a sequence x^hich** covers the various 

aspects of the simplex model. * 



C. Meth'bdology of a Simplex, Growth Model . 

.A > - - ■ ■ . ■ 

Werts, JBreskog^ and 'Linn (1972) discuss models for studying growth which 
require multiple indicator's of the grbi^th variable at each measurement period. 
In this paper a simplex grpi^tW model will be considered wMch needs only a . 

^single indicator at each measurement pertod. Procedures for testing the 
simplex, assumption afi& for -^btainipg traditional growth statistics' are. ' 
discussed. The simplex model appears to be particularly appropriate for 

^ studies of academic ^rox^th (Humphreys, 1960, 1968; Lunneborg, & Lunneborg, 

.1970). J^Jreskog (1970) has provided procedures for, the estimation and 
testing of simplex models. This pgfpe'r will arfalyze quasi-MarKov simplex 
models using a more tecent estimation 'procedure provided by JOreskog and * 

,yan Thlllo (1972) which permits a less complicated and more flexible formula- 
tion. . ^ 
' . - , '"^ 

I . The Quasi Markov Simplex 

^ • ■ * 

The observed scores (y^) are assumed to be relat^ed td their • correspond- 
ing true scores^ (n^O - -fcy the traditional equation: , ' 

4 ^i ^i \^i ' ' ^ ' i ' ' . ^ 

where all ^e independent of each other and o£-all n. :^ Jtireskog 

(1970, sec. 5.6) notes that the simplex structure among the true scores cah 
be stated as* " I* ^. ] • ' 

^^1+1'= Vi'^4+1 * • I' • . V 

where all t,^ are independent and is the true regression weight. As 

noted by Humphreys (1960) , equation (2) implies that the partial correlation . 
between and is zero with n^^j^ controlled. The special charac- 

teristic^ of a growth model (Humphreys,. 196*0) f!s that successive have 

p , 1 

the same units of ;neasurement and th^e difference (A.) between successive 
Vi/ : is given by: ^ ^ 



/^i+i'=/ii +,^1 • f .■ \ (3). 

It follows from equations (2.) and 63) that ' 
11. Estii^tion > * ^ ■ ^ , ^ 

■ — -' — . * : . ■ V ■ 

In-order to estin\^te the parameters of the above mo^l^ k general ' 
tdmputef program (LISREL) for estimating a linear structural equation system 
(jOreskog, M van Thillo, 1972) was use1!f. The following descriptipn is * ^ 
provmed by JOreskog and van Thillo (1972, , pp. 2-4): ^ 



"Consider random vectors n' = (n. 



,n ) and C 



.0 



of tru6 dependent and independent variables, respectivclj^, * and the following 
system of linear s'tructural relations 



(5) 



where B(m x m) and r(m x n) are coefficient matrices and ^^1 ''^2 ' * * ' *^m^ 

is a random vector of residuals (errors in equations, random disturbance terms) ; 
Without loss of generality it may be assumed that. £,(n) = 0 and .^A 

£(§) = 9 It'^is furthermore assumed th^t C is. 4jncorrelated with ^ an^ 
that is ,nonsingular. " . ' * 



(yj^jy3>*- 



The vectors n 
and 



and 



5 are not observed but instead vectors y* = 



y ^= y. + A n + e 
X = V 6 



• ,x^) are observed, ^such that 



where y = £(y) \ V = £(x) and e and/ 6 are* vectors- of erfG^rs.(6f 
measurem6\nt in y and respectivel}^^ The matrices A (p x m)" ahd 



A^Cq ;x n) • are r(*gression ipatrices of 



(6^ 
(7) 



y on n u and of x on ^ , 
respectively. It is convenient to refer to y *^nd x as the observed 
variables and, n 'and- ^' as the true variables. The errors of measurement 
are assumed to be uncorrelated with each other and with the true variates.^ 



Let <^(n\x n) and if^(m. x m) be the variance-couariance matrices of 
and C , resoectivelv, GT ' 



C and C , respectively, 
,-g^rarlances for , y and x 

^^ssu^tions, that the variance-covariance matrix E ['p + q) x (p+* q) ] 
^ = ^r>^')\ is 



and Or. the diaeona?l matrices of error 
, respectively. Then it' follo^^js, from the above 

of 




r^^r'B'""'" + B "^if^B 



A *r'B'"-'-A' 

-x-~ ~ ~v 



+ 0 
~y ~E 



2. 



A B 

~y 



-1 



r4>A^ \ 

V V 



A $A ' 0^ / 



(8) 



-X-'-X 



The elements Qf^^E^ 



•0. 



and 



are \fuhctions of *the elements iof A " A , B. . 

y - -y 
0 . In -applications some of these elements are 



fixecj and equal to assigrted valiies.^ In* particular this is so for elements 
in A , A , B oiind , but! we* shall all(5w for tixed values in the other 
mfatrices al\50. \Tpv the remaining nonfixecj elements ig^tlte six parameter • 
matrices one or more subsets may have identical but untcnown values. Thus, 



elements i^i A 



A 



B 



kin^s: (i) fixed ^parameters thait haye been assigned f^iven -values, {ii) £ 



0^ , and' 



e 



are of three 



on- 



strained parameters that are unknown but equal to one or more other' parameters 
aiid (iii) free patamctcrs that are unknown aiid not cqrlstrained to be .equal to 
any other parameter..'* - \ f * ^ * 



Comparison of equations (1) and (2^ to- the'LISR35L formulae Indicates 
that for estimation purposes A , F , «> , and are not required and may 

be deleted /using a program option which specifies no x. ComparivSon of 
equations (2) and (6) indicates" that A is an identity matrix. Equation 

(^) must be "changed to ~Bj.n^ + n^^j^ - ^t-bl ^° equation (5) format with 

r and' 5. deleted.. The precise form*of B will be shown in the example following 



III, Example o , , 

' .* ^ ^ ) • 

For illustrative^* purposes data reported by Bracht and Hopkins (1972) 
were analyz^ using the simplex -^model. . These^'data include standard deviations 
arid correlations among the composite aichievement scores, fox eight tests in- 
cluding ^the Metropolitan Achievement Test (MAT) a4: grades 1, 2 and 3; the 
Iowa TeQts of £asic Skills , (ITBS) at grades A, 5, 6 and 7; and the Iowa Tests 
of Educational Development (ITED) at grade ;9. Scores are reported in grades- 
equivalent * units for the MAT and ITBS batteries. - • • 

In the simplex formulation:^ 

a. The observed scores at each *grade level ate 



b^ The errors' of measuifemei\t at each grade are 



S' = [e^ > j , , e^ , Eg],- 

c^M The true scores at each .grade are . 

d. The regression residuals ajjong tirue scopes, (specifying nj^'= are 

^.C' -= [Cj^ '^^2 ' ^3 ' ^A. ' ^5 ' ^6 ' ^7 ' ^8^V 

e, . In equation <6) A ^ is an 8 by 8 identity matrix. 



I* ' 
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B = 



f*. In equatii^ii (5) 



1: 


0 ' 0 ■ 
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0 
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0 0 
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0. 



g. The variance covariance matrix, i{; j of t,he * is a tiiagonal matjrix' 



with Entries V . where i = 1, 2, . • 8^ 
h. The variances of the e.« arp 



e2 ' e3 



tollGwing .Joreskog (1970) it can be' shown that V^^^, V g, B^''/^!' y^2.v:„ 



and V^g are not- Identified; i.e., unique estijhates canndt^e obtained. 
Identic l-cati on ,of aM parameters was achieved by arbitrarily assigning J 



values to V , and V> „. 

, el EQ 



C4 



Given these additional" specif loatlbns , there are 8 9 ^ 2 = 36' distinct 



elements in E and 21 parameters to be estimated (6 V^^,' 7 , ^d 8 V^^") 



which leaves 15 degrees of freedom (pveMdentifying res trictiok^) to test 



'"th^ fit of the THodel to the data. 
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The observed variance-TCovariance, matrix (S) for^ the *eight variables 
; in^the Bra%|. and Hopkins (1972) data/is given"' in T^e 1. 



r 



i» » ■ I ' • ■ ^ . ; ^ ■ 

..... " • ^ ■ . ^ ■ , . • ' 

' . • ^ Table 1. Observed^ variance-covariance matrix; 

\ ■ ^ , . -fe^ 

'•0.260. '■ ^. ? ' ' ' • ' X 

^257 • . 0.476 ' , 

0.336 'fc 0.528 "0.792 ,', > , ,- ' ; ' _ 

0.371 . 0.551 0.782- 1.020 /• , . 

0.416 d.646' ■ 0.918 1.127 l.'44Q --> " ;, . ' " " 

0.437 " 0.661 0.'-942 »1.158 ' 1/406 / 1^588i < , v 

0.465 0.7Q5 : '&.995 1.2i3 ' l."490 . 1. 634 1. 9044 , . 

1.576 • 2.168* 3.2dJ8' 4.005 -5.00'6 5.516 •6.Li'2 2-6.523 



LISREL yielded the following maximum-likelihood<j parameter estimates: 



• 



a. ' Parameters yi B : B^,- 1.318, = 1. 054, ' B, k ijl^l, B^ « a.*D26, 

- 1.055 arid B^' = 3.367. ^ • . T ' 

b. ■ Paramet>?rs in ; = .049^ V - = .137, "V. = i052, V = 

. • . ' V = .091.^ S ' ^4 ■ . ^5 J : ' _ 

c. -^' Parameters in - 0^ : V = (.276)^,- V „ = (.222)^', Y , =' (.240)^," 

.V^3 = (.260)^ V^^-= (.193)% V^^ = (.298)^? \ ; 

Parameters not liste4 are not identified. The program alj5q estimates the 
varl^fice^covariance matrix among the true variables (n.).r 



A ciruclal part of the outp^it is the estimated value .ofKE dnd the cor- 
. , ■ responding discrdpanciea from the observed variance-coyariance matrix, S-. 1^ 
these* are so latge as to irvdlcate an , incorrect rttiodel, then the above parameter' 
estimates woyld ha^ little meaning. The resi"dual^ of ? - i are given in Tab],e 
^ - 2. Because of the difficulty in comparing residuals between variables with 

different variances, Table '2 gives discrepancies after S and Z have<becn stahd- 
ardizpd to qorrelation matrices. Considering tlve fairly large number of ever- 
identifying restrictions (df'v^ 15) and the fact'^.that S is, a missing data matrix 
with sampla sizes ^ranging from 300 'to 1240, ^hese results indicate a r^asonalily 
good fit of^ the^Wodel to the data. The gopdnfess of fit test assuming a sample 
size D^f 300, yielded a chi-vsquai;e of 26.3 with 15 degrees of freedom. TJia ' 
^ probability of getting a chi-squared value larger than fhat actually obtained^ . 
gdven that- the liypothesized model is true is = .035.^ Because of the 
relatively, Targe samples this s'tatistic is of minimal interest because quite . 
small discrepancies will be statistically significant. . ' 



r 



■ '4 

tabiev2'. Resi-rltials S. - j; ,s taiidardized . ' . kJ'^. 



-0.000 ° 



-0.000 ' o.oo"d .-7* • ' ' ^ 



-0.'.006: 0,00l. -0:000; -\ 

0.027''- -bMl'- -OiOOl^ . -0.000 

-oVoo3v ^ -o'.oos 0.001 .■• ^ o.'bob. ., p.ooo ' ^ ' 

0.013^ -vO.Ob? ' O.OOl 0.00], -0.001- ,0.000' 

0.018. 0.002 0.003 -0.005 . 0.004 f -0,000. "-0.000 



0.021 ' -0.656 -0.(J28 -0.019 ~* 0^.001 0.002 -0.000 ,^-0.000 
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7 To facilitate interpretation of parameter estiTnates\a program option^ 
was iised ^^yhich standardizes all p>arrameteir estimates and provides the correlation 
among th*e 'true variables. / Thiese re^sujts er4 shown in Figure 1. Werts, JBi^e^ko^, 
& Linn.' (1971) have sho\m rhat the correMCion betvjjeen y-j and n2 between 
yeaind n? are Iden trifled (festima ted at .797 and \ 881 respectively). The 9 . 

estimated relia^jillty -of/ each* observed -variable 4g the square of the correlation 
with 'the corresponding trye score e.g. the reliability oJE y2 is (.916)^ ^ A.B39.. 
Ifhe correlation be&i7een any "too nonadj^cent tariie variables^ is eqifal to the 
product of the interveAing B?. 'Alie s-tanda^dize^a B*. (B^) ate equal to the 

unattenuated correlations (a. e. corrected for unreliability) between the 
corresponding variables in. th^ ThodGl* • The average reliability for yi through^ 
yy is •estimated as .93 which ,qj3ropai>es w^^th *98 reported b^ Bracht & Hopkins. 
tl972) . Our lower estimate' iiiigkt ari's^. (a) bec^uS'e of ,noniudepe^dent /\ - 
errors af •measurement for the various ^subteis each composita» resulting (v^^ 

in overestimates of relial?i;^lty by the procedure employed by ^Brach;t^ Hopkins . 
and/or (b) because estimates cietived f rom structural models involvings / 
theoretical proposltiops , can ;be expected to' reflect hpth reliability and * 
validity. tyj^e measuremeHft errors, and/or (c) because the model does^not fit * 
perfectly. " ' ^ , ^ ' ^ - ' _ ^ . 
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IV. .GROifJTH statistics . ' • * 

... ' . • - ■? . 

• 

A growth model requires variables to^^ have the same units of measuremeW 
over time. This' is necevssary iU order for the difference bet\^;een final and 
initial status toVbe meaningful, i.i6. subtraction on]>%makes scn^e wli^n tPie 
u-nita'are the same. In -the example analyzed in section III the pcoroa for - 
the MT and ITBS' batterieV wfere reported in grade-equivalent units f Whether 
the units are in fact equi>?^alent»^er is unknown, ^iov;evcr for. Illustra- 

tive purpos-Ss' tluey will be* sO treated. ■ f\\ ' ^ ' ^ * / 

.The variajlce-covariahce matrix for the* /^rue feactoiTs .( ^l?) is the hasdc 
datum for computation 'of *growth statistics.' 'iJerLs,' Jilr^rskog, ^ Linn (1972) 
have shown that: a. th^ variance of t^e gain scores, may be estimated^ 

frdm V, = VL. + - 2 C (n^ n.^^^) . ^ 



n:. 



n 



i+1 



X9X 



b. 



where C(n^ ^i+1^ covariance between n,^^ and 

the true correlatlpri of status x^rith gain> 



p(n. & is given by: 



c. 



the reliability of gain scores, 



i 

P/ 



■A. 



+ V 



+ V 



-1+1 



(10), and 

is given by: 



The estimated .varianc'e^covariance laatrix among the true variables, is 
given in tablets Except for the unidentified variances of m and t\q, > 
For this 'reason no growth statistics involving m and r]i can be ^onp\ited. 
Comparison ,of tables 1 and 3 t^ill show thaf the coyarlances be.tween the \ 
true' variables approximate thosd between the correspondLrig observed variables. 
If the model is correct any discrepancies would be ascribed to sampling 
errors since according to equation (1) the observed and true covariances 
should be identical. • ' - 
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Table 3. D^ariance - cQvariance Matrix |AinonB True Variables^ 



s • 






. 1 " - 
















^5 


He 




0.257 


0.400 


\ 










•0.338 


0.527 


0,74^3 












'^555 


0.783-" 


Q.-962 • 








* oT^isV 




i 

0.917 


1.127 ■ 


1.372 






''6.428^ 


0.667^ 


0.940 . 


1.156, 


1.407 


> 550 . 




0.452 ^ 


0-.703 


0.992 


1.219 


.1.A84 


■1.635 


1,815 


1.521 


' 2-. 368 


' 3.339 


4.104 


4.997 


5.505 

,1 


6.112 












; ( 





*Not ider^tified 



The results in Table 3 x^7ere used in equations (9) , (10) and to compu 

the growth statistics presented in Table The average true correlation of 
status witfis^gain is estimated_to be .39 (Fisher's Z transformation used for 
averaging) . l^The average reliability of gafn- Scores io estimated to be .46;i:e 
.the oi^<Ier of magnitude of the true change variance^approximates that of the 
associat(^d errors of measurement (0^). TaJ^le 3 could be used witli equations 
(9), (10), and (11) to. comjpvite change statistics betwcon any twb true Arariable 
Q.^. the n? • n2 period yields a change variances of .809, a correlnliwi ofs. 
status witl> gain of ;533 and a gain reliability of .831. The meani/gfulneffVS 
of these statistics is dependent on thef correctness oi the assumption of 

units of ^measur^ent over time. ^ a 



te 
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V* Table 4. Estimated Growth Statistics V 



Change 



Clvange 
Variance 



Status - Gain 
torrelation 



E<stliabildty 
"of Gain 



tl3 - Tl2 
>'l7 "1 6 



.089 
.139 

.081 

.108 

.09^ 



.31 
.054 

.171 

.026. 



.055 

I 



/ 



.674 
.125 

.589 

.093* 



i 



. >415^ 

■';;*565 

9 
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V. A Structural Model for Grox^rth 



* Tlie above estimation model used "^Jnly' the simplex equations (i) and (2) . 
For estimation purposes equations (3) and (4) could have been used directly to 
define the model. -It follox^d from equatiouc (3) that " 



1 ■ . ^'u 

C case the vectar df true variables becomes: 



'i+1 



m + K A . in x^ihich 



= \(ni, Aj, A2, A3, Af,, A5, Ag, A^) and 



,y , e , and ^ remain the same as before. Equation (.1) becomes: 



y = (m. + E A ) 
and equatiori (4) : 



(12) 



A, = (B, - 1) (n. + E A^) ^ 



Trans;!^atiAg these equafions into equations (55 and (6):, \ 



^0 and', ij^ i^main the same. .It follows front equation (12). . 



in the example: 



Aj yA = 



r 



1 


0 


0 


0 


•Iff 


0 


0 


0 


1 


1 


0 


11 


4 


0 


0 

« 


■ 0 


1 


1 


1 

X' 


0 


0. 


0* 


0 


0 
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1 


1 
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0, 
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l" 
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0 
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0 
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1 


1 
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1 


1 
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0 


•1 


; 


1 


1 


.'1 


1 


1 


o» 


1 


1 


1 


1 


1 


1 


1 


1 



TrMisla^ing equation (13) and defining b. - (B . - 1) 
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0 0 0 0 0 0 0 



b. 1 0 0 0 0 0 0 
^2 ^2 ° 0 0 0 0 



"3 "3 S ' 



0 0 0 0 



b, b, h\ b, 1 ' 0 0 0 



b3 b^ 1 '10 0 



b b b b- b b 1 0 
"6 6 6 6 6 6 . 

b, b^ b, b, b, b^ b^ 1 
„ 7 7 7 7 7 '7 ^7 
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This model, is simply a linear trans formaLioti of the previous nodel, has the* 
'^ame* number of pardmete-^-a to be estimated (V & V arbitrarily fixed), and . 

the model should (and did) fit the data to the same, degree an in the pre/ious 
analysis." It should be, noted however that this formulation required use . 
■ (in B ) of the LISREU option permitting. parameters/ to he specified as' equal. 

Tlie ^ariance-covariance matrix of the fef'rue variables (C) in the growthv^ 



formulation yields the estimated' true change variance 



A 
' m 



directly 



''It is reasonable to use this fprmulatipn in conjunction with the previous 
formulation to obtain the change* statistics . Also o'f' supplemental interest^ 
is yfili»st<ind^dized C which gives the correlation among the change factors 
as sHoxm in T^l^s^ (Aj £j.nd A7 are not identified). , It; can be seen that most' 
of the correlations are l^pw positive, Diffej^ces between these, ctor^elatione 
are difficult to interprett becaus^ ''of fluctuation in the associated fchange 
variances. ' , ! ' ^ ^ / 







J. ■ ■ 










Table. 5. 


Correlations Among True 


Change Scdres 


'■ : [ 
















A2 


[ A3 ^ 


A^ 








i.ooo ' 






ft 




A3 


. .104 


1.000. 










.460 


.290 ' ■ 

> 


1.000 • 






As 


.069 


.044 


.067 


■ 1.000 






.162 

! 


. .102 


.156 


.077 


1.000 



19- 



ERIC 



^18- 



r 



■ VI. Estimating True Grov?fch , ■ ' 

Werts, JBr^eskog and Liiin (1972) give a ^procedure for ostimatixig true 
growth scoreB from all observed taeasure^ in a* structural model. TJie basic 
» p3:oblein is to obtaiiK^ Variance-^covariance matrix for thQ observed variables 
and the true clj'migey^^ICo be predicted e.g. take Aij ^ TI5 .n^ the variable 
ta be predict^ti. follows from eq»ataons (1), (2)^ (3) and (4) that the 

covariance' bi^ tureen the'^y, aft& A^^ are given by: o 

. ' c(yi At,) = [Bi, - 1] B3 B2 cCmna) • , 



C(yz = [Bn - 13 ,B3 B2' V 



C(y3 = [Bi, - U B3 f • ■ 
s,C(yt, At,)' = , iB'i, J- 1] V 

C(yJ^ A|) .= [Bt, ^1] V^^+ V^^ 
cCye.At,) = Bs^ C(y5 A,^) - 

CCy? 44^== Be C(y6 At,) . .» 

cW^itlfv^ C(y7 At, J, . ^■ 



^ For t^e purpose of estimating Ai^ it is appropriate ^to use tshe, estimated ^ 
variances and covariances among the y^ as obfain^d from £ pather than the ^ 

obiserved matrix because ^ is presumed to be the best popiA^ion estimate 
of these values when the model is accepted. Tlae resulting Variance-rcovariance 
m?itrix is. given in table 6. -standardized form to facilitate interpretation. 
• LISREL was then set up for the regression of on the y^ii^ The y. predicted 



|&3%'Of the variance in Ai^, "This is not much lart^er tlian tlife 59% of the 




ERIC 



{ 



Table 6. C(Jrre rations Among and Ai^ 



4 


1.0X)0 • 


y\ 

! %• . • 

f 










\ 




.421 


1.000* 


# 


• 


* 






72 




.731 


1.000 


• 


( . 






"73 


.529 


.'i'47 


' .-858 


1^000 








y^ 


.572 

\ 


.693 


^797 « 


.^71 


1 

■ 1.000 


V 




ys 


i'683 


> .785 




a". 9^ 


1.000 




ye 


.702 


.•666 


.7^7 




.908 


.930 


1.000 


V? 


. .677 


.642 


.738 


.808 0 


" .875 


..896 


.940 


ye 


• 


,5f 


.666 


.729 


.789 


.809 


.848 



77 ' 



ye 



1.000 



.860 1.000 
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variance predic^table frprn^y^ and alone because Reliabilities of 

and y^ are quite high./ Note that tKe reliability of .39^13 not r 

directly comparal^le to these figures because reliability corresponds to the 
;jpToportion of observed variafice predictable frqin the true score whereas our 
* procedtftre ±^ the reverse. , ' 



VII^ Discussion 



stucfies of Academic gi'-owth i t -is ' ii&eful to. detail, the rationale behind the 
es tiination^^ the reliabilities of each time, yor, exagiple consider the' 
reliyability foryy^ i.e.. the squared correlation of and ns • 

Given -on^ measure prior to the fifth gra^ (m =' 1,2,3,4) * and /one measure 
subsequent to the fifth^ gfade (n = 5,7,8) then in the simple/ model -'the ■ 
reliabili^ of y^ Is' - , 

^ - ' p (ys T^s)^ 



In order to urMerstand the value bf the quasi Markx)v simplex^ model for 



p (y^ ys) p (ys y^) 



p (y^ y„). • • 

\^en the model fits the dat.ia the implication is that except ,for sampling 
variations the estimat?fes of p (3*^5 TI5) deprived from the various combinations 
of y^. and y^ will be equal. The greatei: the number of consistent -estimates 

(m X n - 12), the Sbre gcneraliy.able the results are likely to be. This 
method contr^asts w^h the split haitf/or parallel forifi methods of obtaining 
test reliabilities v/hich provide a single estimate which cannot 1)e rejected 
because of inconsistency with the data (i.e. it is "ju-st'' identified) . The 
split half or parallel form proceduries involve almost identical item formats 
which could well lead to overestimation of reliability because of the V^esence 
of ''method" variance, Thg simplex model apFiroach is less subject to method 
variance because over time both item format and content chaTlige. This is ^ • 
probably the main reason that in the example the reliabilities from the 
simplex model were less than that reported in the data source. ^ 
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D. Inlpl 



[plications for the S\udy^jf\ Growth 



Thorndike (1966)' notes that in order^to accurately estimate the 

*' * ■ " ' ' \ 

correlation betwee]^ initial intellectoial status and saabsequenft intel- 

■ ' U \ . ' ^ ' ^ 

lectual growth it is necessary ^to have measnjres expressed in tegning- 

• ' - • ' ^ ' ■ / 

fully equal units which at all sjges .refer to identically the sairife 



attribute of the individual and to eitiier have* error^free 

■ . 



measiures 

r 



or accjurate reliabilities. Because error-free measures are nat 
available coxrections are typically made usiagay^ilable reliability 
coefficients. / however, as . Lord ,0^963)- Has^-tf^^ thfe need to make 
corrections for attenuation .poses isomewfitkt of 'a dilemma, since, 

first, if is oftlsn hard to obtain the particular kind of reliability 
coefficients that are required for making the appropriate Correction, 
and the correction"... may be seriously affected by sampling errors*-. 
Because of the fragility of gain scofes^^^it is particularly imi^prtant 
t<p have an accurate reliability estimate. As pointed out by Campbell 
and Fiske (1959), procedures based on similar measurement methods 
(e.g.^ the usual internal consistency or parallel form reliabilities) 
will be biased because of method variance (i.e. correlated measure-, 
ment error). In' this paper unreliability estimates will be generated 
using Of procedure less subject to method variance whicll is based on 
the simplex model (Humphreys , 1960) . 
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Method. - • ' " . ; ' • 

: The analytical procedures used la, this paper'^are detailed In ' ^ 
Werts, Linn, ancf. JSresUog (In press).' A geileralj cpmputer program 'f§r 
estiiuatlng a linear ^rubttiral equatloi^* sys.tera Involving multiple 
indlcafcdriS of unmeasure'd variables called LISREL (Jtireskog and 
van Thillo, 1972) ^tsTas 'u3^ed for all cdinputatlona* ' • . 

The basic model , iiseia in the analyses is called a "quasi-Markovf 
simplex" by Ji^reskog (1970>. Each pb Served test score (X^) Ms assumed 

. V . • ' • , ■ ■■ . • - ■ ■ 1 ■ ■ 

'to consist of a true component (T_^) ail^ an Independent error (E^'V.- ' 

All of the E a;|^ assxlmed'independent ' and successive T. are t^elated ' 

. ^ • - If . , • ■ ^ • . ■ 

■ ^ 

Hy the llneai*^%gression equation: 

where the d_j^ residuals are Independent. The reliability t^^,_^) of 
an observed X. is equal to (n<l,(m>i): 



^11 in Im nm 

where R, is the correlation of X^ and a prior measure X R. dts ^ V 
in ^ ,1 ■ " ■ - .n, im ^ ^ 

. i ^' 

tHe correlation of X. and a later measure X aitd R is the correlatl 

1 nly nm ^ 

of X and X • If th^re is more than one observed score prior to oir 
n " m , 

following X. ^ then there will be more than one possible estimate of 
R^^ If the simplex" model fits the* data, then all possible esi^tlmates^ 
of-R^^ will be equal within the WMts of sampling error. It follows 
from eqtS^tion (3) that reliabilities cannoL be estimated for either 
the first or the last measures, ^When successive measures are on the i 



same scale,'* then changes.. In successive true scares (A: )* over time, T 
kxB defined by: ; = "''l+l^ '^l^ I'or • analytical purpoaes\^estl^lates 
'*of- .the variances (V^^) o f the . the . true * regression weights (c^lT) , 
the reliabilities (R^^)j the true change, variances (V^^) , and ^ the 




time crfrrelakpn Qf Stratus .w^ (Pt^a-j^ ^^^^ be'-mo^t relevant,) 

Data ' ' : . , r 't, ' • 

The d^ta JEor this study were coiUec'ted in a t,en-yeaf longitudinal' 
study of aGademic growth at Edticational Testing Service (Hilton,' Beatoni 
an^ Bbwer 1971). > ' ' ^ \ 

, . ^ \ ■ • 

The School and?);Collegjer Ability Test (SCAT) and Sequential jTest 
r r. ■ i • V - „ ^ ^ . , ,^ ■ * ^; ; 

of Educational Progi^s "(STEP) were given in the fifth, seventh, ninth, 

and eleventh grades. StAT was designed to meastire basic verbal' and 

' ' ' ■ ' ■ • ' • ir ■ ■ 

c^aintltatlve aBill ties and provides 'Verbal, Quantitative and Total 
scores, STB? was designed to measure skills and problem-solving 
abilities which are generally considered major goals p£ educations gnd** 
fields six subtest scoresV Mathematics, Sci ence. Social Studies* 
Reading, Listening and Writing. The analysis was done on the 2,483 

■ . . . ' ■ ■■ • . ■ ' ' ;i ■■■ ' ■ • 

students for whom SCAT and STEP^ sconces were available foi: all four 



grades. The variance-covariance matrices for these teFt scores is' 
giveii' in. Table 1 in which the first four columns give the variances 

for the four occasions and the next six Columns giVe covariances 

^ ■ ■ ■ • _ %• 

between these occasions, >^ ► 
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Testing the SlinplexVjFit For Edch Test . 

The most crucial^^^rt of the analysis concerns the fit of the . 
simplex model to the-4axa. If th«? model Is Inconsistent with the data, 
then paraiiBters estimates *are meaningless. Jtireskog^s LISREL program 
provides a large sample chi-square statistic foi: testing the "fit of 
the model to the data. In essetice, this chl-square is a meastgre of 
how close the "reproduced" matrix is to the observed varianced- 
covariance matrix. The "reproduced" matrix is the matrix generated 
by the estimated maximum likelihood parameter estimates. -^When sample 
sizes 'are large, as in this case, quite small differences betxireen 



the Observed and Reproduced matrices will, be statistically significant. 
To help judge the meaningfulness of these differences, •both matrices*, 

" ^ , j^' 

were converted to correlations and the root mean, square of^ the * 
differences between the observed and reproduced correlation matrices 
was calculated. ■ ^ 

In Table 2, the chl-square is given for each of the tests. 
The Science and Reading subtests show a statistically significant 
lack of fit as seen from the significance levels given in the second 
column. Howeve%" the root mean square dif ference-^ie^een the corre- 
sponding observed and reproduced congelation matrices is only :.005 
and .004 :^espectively. Such small differences are clearly not 
mearil|igful. We conclude that the simplex model provides an e\cellent 
fit to the observed data for the four occasions. 
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TEST 



BASIC X-' 



BASIC P 



RMS Reaiduals 



STEP: 

Math * 0.17 

Science 5.39* 

Social S. 3.09 

Reading . 8.06* 

•* ^ - ' 

Lis-tcning 0.53 

Writing 1.06 
SCAT: • 

Verbal 2;. 18 

Quant. 0.24 

Total 0.00 



.683 
.020 
.079 
.005 
.468 
.303 

.140 
.623 
.973 



.001 
.005 
.003 
.004 
.001 
. .002 

.001 
.001 
.000 



Table 2. Chi-square Goodhes 




t Tests 



*Significant at the .02 level or better when the x is tested with 
one degree of freedom. 
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^ Egtimated Gtowth, Statistics , 

Using formulae from Werts, Linn, and Jtlreskog (in press) various 
growth Statistics were' estimated, Since^ the reliabilities for the, 
fifth and eleventh grades are not determined by the model, it follox^rs 

that V^^^ V^^^ B^^ R^^^ S^^^ V^^^^ V 3. -PtIAI. ^13^3 

be eD.tlmated. p * 

Parameter estimates ^for each test are giyen in Table 3, It' can 
be seen that the true variance increases from the seventh to ninth 
cgrafles (i.e., ^° ^ij;3^ tests except ^^ing. In inter- 

preting the seventh to ninth (B2) .and ninth to eleventh (B^) grade 

true rejgresslon weights, it^shoul'd be noted that a^welght of 1,0 

I 

means a zero correlation of true status with true gain, a x^eight 
greater than 1^0 means a positive correlation, and a x;eight less than 
1.0 mean^ a negative correlation. B2 and B^ x^ere tested to see if , 
. they were significantly different from 1.0 (Werts, Linn & JHreskog 



in press) . These results (Table 3) Indicate that B^ is less than 



Reading gnd Is less than 1.0 for Science, Soci"al 



1.0 only 

Studies, Listening and Writing. The signiflcan^ce test for B2 - 1-0 , 
significance level for ^^^2^2 l^st column of Table 3. 




The true change variance from seventh to ninth grades V^2 
compared to the sum of the error varimces for the tests at thes%^ 



times (la^ielled V^^^ in ^Table 3) . 
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J 



The reliability of these gain scores would be 7^2^ ^2 \e '^^ 
can be observed that the true variance is quite ^mall compared to 



the error variance for the STEP subtests but is inuch ,niore compar- 
able for the SCAT subtests! This is a function of reliabilities' 
ai}d merely points out that obtaining accurate change statistics is 



, possible^ ojily with highly reliable tests. 
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Methodolog;ical Consideratiofts ^ V 

• As noted above the true chahge variance Is typ|caHy small 
compared to the observed variances or the estimated error -variances 
This means tl^t accurate cecrrections for atteinuation are essen^tial 
since a small o,r moderate error ii^ estimating unreliability will 
normally res'tilt in a relatively large error in estimating true 

5 . ■ * 

change variance, . The xisual procedures for estimating reliaBility 
Itivolve split halt or parallel • form methods which invol\(fe almost 
''identical' item formats which could well lead co overesiimation of 
reliability because some of the item covariance is due to J'method" 
variance (Campbell and Fiske, ^1969) . Thfe simplex model approach 
is ^ess subject to method variance bec^se over time both item 
format and contient change. - This is probably the 'main reasotl that 
reH§bilities from the simple^ model are usually less 'than those 
reported by the test makers. It is not unlikely that many studies 
of Jttie determinants of academic growth Cor change) failec^o find 

correlates of change because of inadequate corrections for unre- 

/ - 

liabil^ityi , ^ ' 

The ffrst crucial step in any study of school effects is to 
measure the changes or groxijth in cognitive skills during the period 

0 \ 

of interest. Iti other Xijords it is necessary to Icnox^ precisely a 
person *s skills at the sta\jt and at 'the end in order to specify 

r 

what x^as learned during the period. Thom^like (19661 pointed- out 
that this ^requires having the initial and final measures in 



nieaningfiilly eqjaal units which refer to identically the same ^ ' 
^tribute of the individual. The logic of this requirement is 
simply ehat if the final score is 7 pears and the initial score 
is 4 apples we can neither specify the number of pear^ npr the 
number of apples gained during the period. For example, if the 

- - , . . \ 

final t^st mead'ures reasoning ability and the initial test rote 
memory it will be impossible 4:o know how much either ability. has^ ' 
progfess,ed in the interim. If tfte final score is 7 large apples 
and the initial score '3^ small apples, "the %hange is at least^A 
large apples.. However, if the final s'core units 'were small apples 
and the 'initial score units large apples , the gain i4 difficult to 
specify. Thus, even if accurate corrections for attenuation are 
possible, growth may b^ easily, obscured by problems of scalin-g the 
units of measurement over, time. <• ' 

The results in Table 3 should make it clear that the observed 
correlatl-ons can have a simplex -pattern x^heu the true corre^latidu. of 
status with gain is positive, zero, or negative. Assuming independent 
errors, if the true gain is * unc^related with true initial status,* 
then the observed correlations will haVe a simplex pattern. I^t 
does not follow, however, that if a simplex correlational pattern 
is o|?served that the correlation of atafus v/ith gain is zero as has 
been sjiggestek by Humphreys (1960), Andersen, (1939), and Bloom (19&4) 

Furthermore, the results in Table 3 suggest that there may not be 

^ J % 3 ... ■ 

a single true correlation between intellectual ^tatus- and intellectual 



growth. The status gain correlation -of +;09 for SCAT Verbal and 

♦ 

+.37 for &CAT ^Quantitative based on quite reliable- tests , suggest - 
that learning quantitative intellectual skills' may be more dependent 
on prior learning than verbal skills. 
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E. Analyses of Longitudinal Grade Data 

r , 
•i" * 

Humphreys (1968). notes that the correlations ^mong eight ^tnesttirs of 
undergraduate grade-point averages have a simplex form* By this he means 
. that the farther apart the averages are in time the lower v/ill be the tor- 
relation between them. ' Furthdtmore, Humphreys (1960) notes : . "If ^onc is 
sxi^f ficiently confident that the variables do form a simplex, ^ reliability 
^ estimate can be obtained from the inter&orrelatlons of ^ the va:(:iables." In^ 

*' ^ i • 

this paper a procedure developed by JBreskog '(1970a) will be used to test 

1 - 
. <5 ■ . * . 

whether Humphreys^ (1968) college grade data form a simplex and to obtain 

f» » ' . 

estimates*\of reliabilities and unattenuated correlations between grades. 

/ - ■ ■ ■ ■ " • • y 

I. The !ltf.d&l ° . " ' 

0 The model used by Humphreys is called a ^'quasi->larkov simplex" by - 
JBreskog (1970a), In this modql each observed grade score (X.) is com- 
posed of a true component (T^) and an independent error (e. ) df 
measurement: ^ - « 

h = + . . , ^ (1) 

All of the are assumed independent and successive T, are' related by 

the linear regression equation: • 

Vl= Vl-^-^i ' ' . / (2> ■ 



f where the d^c^ residuals 'are assumed independent of each other. 

In diLs model ^ the reliability (^-ji^) of observed ^'X. is equal to 

(n<i> m>i) : ' o 

r. r. . • ' . 

in im ^ 
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where r. is the correlation df X. and X r, is the cjorrelatlon oj: X. 

in ^ ' . i < n , im i 

*■ , ■ • 

and X and r is the correlation of X and X If there is more than 
m nm ^ . • n m. 

onq observed score prior 'to or following X then thqre will be more than 

« i , 

one poss^ible estimate of r, . If the simplex model fits the data then 
all the possible estimates of r^^ will be equal witkiii^ the. limits of 
sampling error. It follows from equation (3) that reliabilities cannot 
be estimated for either the first or the last observed measures. 

Jlhrieskog's (1970a) procedures for the esLimation and testing of simplex 
models x^^as used. This method provides a chi square goodness of fit test 
and also shows how well the estimated parameters reproduce the observed 
correlation matrix. The details of this analytical procedure are ' 
beyond the scope of thi^s paper, . ► 

il. Analycis 

The correlation matrix i^own: in Table 1 was obtainp.d from Humphreys 
(1968, Table 2). The variables include eight semesters of grade-point 
averages, high school rank, and composite score on the American College 
Testing program tests fibr approximately 1,600. students at the University of 
Illinois. / ^ ^ ^ . 

^ Simplex Model for Eight Semesters Grade-Point Averages . 

The initial anr^ysis was designed to test ' the 'fit of the simplex model", 
to the correlations among the eight semesters grades. The goodness of fit 
test yielded a chi square of 23.91 A^ith 'iS degrees of freedom. The probabil- 
ity of getting a chi^quared value larger than that actually obtained, 

as .' 
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# Table! 

Correlations among Obser^d Variables 
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l.OQO 



Note: Xq Is high school rank,* X* .ACT composite score, and 

X^ through Xg "are eight semesters grade-point averages. 
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>tVi^^hat>c hypothesized model is true^ is, P = 0.07 . Especially con- 
sidering the large sample size, these results ara consistent with the 
hypothesis that-^the simplex model provides a g'ood f>^.^|;the data. If 'the 
simplex model is rejected, est|mates of modfl garnmetS' would not be 
relevant. _ The estimated param^rs shown in Figure 1, include 
correlations between X. and T. and the correlations among the t/ . 
Reliabilities are equal ta the square of the correlation between ^ the cor- 
responding X. . and T^- : e.g.. •r22 = (,75/0^= .569 . Although not sho..ni, 
the correlation of X^with is . 737 and of Xg .rtth 'is .69^. 

The maximum likelihood estimates given in Figure 1 could be" used to gener- 
ate the correlations among the observed variables; e.g. , r„^= (. 754)j(.838) (. 758) 
W»79. Tlie estimated correlations generated in this manner differ from the 
observed correlations only because of-sampling eri^rs, if the model, is 
correc^. The estimated correlatioas 'are therefore estimated population 
values giverf the simplex model. ' In Table 2 the discrepancies betr.ecn 
the observed^and the estimated correlations are shown. The small size 
of these discrepancies is consistent with the chi square statis tic 'in 
suggesting a good fit of the mode? to thfe data. Unlike the chi square, 
howeyer, the discrepancies do not incrLse as a function of ,the sample 
size. ^ . ' ■ a 
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'''r.uro 1. Slmprex pararaeuer estimates 
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Table 2 . 
discrepancies betxjaen Obs.erved and Estimated 
Correlations among Observed' Variable?; 
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B. Af^su m ption of Kqual Rellabilillofj 

The data led Humphreys to believe that the reliabilities across / 
semesters uere equal> JBrc&kog's (1970a) procedures allow this hypothesis 
to be .tested because modfel parameters may be constrained to be equal. 
Because of special fea"^ turds of JSreskog^s simplex analysis the reliabili^ties 
could not be directly constrainedj hov;evef the same effect vras obtained by 
setting the error variances equalo As a result of ^hesQ constraints the chi 
square increased to 26.08 x^^ith 20 d^,rees of freedom. The increase in ^hi 
square of 2.17 (i.e.,, 26.08-23.91) with 5 denrees of fTcedom <l.e., -20-15) 
is an appropriate test of the^ equal reliabilities hypothe^ris. Since the 
probability of obtaining a larger chi square is approximately .80 this 
hypothesiswas not rejected. These rcsultr> therefore support Humphreys' 
conclusion th^t the reliabilities are equal. The estimated reliability is 
.579 x-7hich corresponds to a correlation of .761 bet\7een arid T^ in 

Figure 1. Reading from left to right in Figure 1, the ncx7 true correlations 
assuniing equal reliability, are .836, .965, .891, .936, and .922. Tliese 
differ very little from those dn Figure 1. If it is assumed that X^^ and 
X:^ have a reliability of ,579 then the correlations of T^^ and T^ x^ould 

be .966 and that of T« and T^ .914. 

/ o 
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The computer prograia iioed ^or'thte autolysis my be used for a wide 
variety of other structural jnodelG (JBrcskoR, 1970b), sone of v;hich laifiht 
include the Giraples: as a- component part. To illTjir^trate thin, a jnodel will 
be hypothesified vjhich includes high school rank ^ (X^) and the ACT eorap^i:Ute 
score (3>^) . It searis rc>asonable to suppose that both these vgrriables are 
measures ^of high school achievement, i.e./ ' * 

0 

If high school, achievement also is part of a simplex pattern then can 

be included in equation (2). As before p all errors (IncKiding f:^ and^c^) 

are assumed indepenclent . Building on-prior results it will be a.^sumed that 

the college grades hav,e equal reliabilities. A f^pecial iealure of this inf)dr:] 

is that the reliability of X^^ can be estimated from -equation (3) usinp 

either X^ or X^ as a prior variable. The analysis of this model yielded 

a chi square of 45.22 with 34 degrees of freMom. Since the probability of ^ 

a larger chi square Is .095, thes*i results suggest thpt the" hypothcsi?ied 

model is consistent with tfie data. The discrepancies between observed and 

estimated correlations are on» the' order, of those given in Table 2. The 

i 

estimated parameters are given in Figure . 2. Tlie reliability of college 

grade-rpoint averages is estimated as .584 \/liich does hot meaningfully differ 
from previous ^Jesuits. Comparable^ correlations among trye scores also 

differ little from Figure 1. The estimated reliability of hig^ school rank 

is .424 and that' of the ACT composlte^ . 365. The ACT composite teUabiJitv 

is substantially lov7oi^ than would be expected if parallel forrn or internal y 

consistency .estimates were obtained in which case a value closer to .9 
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,iai8ht be eispected. Tlie reason for this low value may be Lhaf ACT 
ji,- . * 
composite was the only test score included in the model. Vnuh^ thefe 

may be substantial systematic variability ine^ but it is simply un- 

, corrQlatQd with the grade true scores. If t^^o ACT composite sdores wete 

available for each studeht v/e ^ght postulate a model such as the , one 

55hovm in Figure 3.^ According to the mo-dol in Figure 3 the observed ACT 

scores would be represented by " V * 

ACT^ T + S e ■ . 
i o * o 1 , 

• ♦ 

and ACT = T S -I- where g- and are uncorrelated with each other and 
with.T^ andS^. Further, and are uncovrelated. With the above model 
the reliability of ' the ACT comj^osite might be xloser to the expected <value. • ' 
This modification of trhe model, however, would not l^xTto changes in the ^ 

o 

I ■ . 

other parameter estimatec of the mo^l shown in Figure 2. - 

- ■ . ■ > 

The reliabilities shown i«r Figure 2 also may be lovjer foi" this- sample 
of students who have completed eight semesters of college than It would " 
be for a full range of high school students. The results are however 
'c^nsistenlJi w^h the hypothesis that high school rank and ACT composite 
. measure fthe san^So-^ue variable and that -the simplex model fits high school 
and college achievement < 
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Figure 3. Fpstulated Model for HSR and 
Two ACT Composite Scores 
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. D. Additional Analyg^s ' 
^ Humphreys and Taber (19'73, Table 1) provided correljatloi^fj among eipjlt 
\ semesters of college grades among seniors at the University of Illinois 
x-jho' took the Graduate- Record E>5aminatJon» Using a simplex model assuming 
equal reliabilities a chi sqtiare of 43. 5* with 20 degrees of freedom wan 

» • 

obtained. Although this is statistically s^ignificant at the P - ,002 level 
the" fit as judged from discrepancies betvjeen the obsdrved c^jf relations and 
those estimated from the model was close to that shovm in Table 2. In 

' part the poorer fit might have resulted from, the fact that those are 
missing data correlations with sample si^es ranging from 1549 to 3018. 
The estimated reliability of .683 is somewhat higher than that previously • 
estimated. It is interesting to note that the ratio of thenc reliabilities > 

J i.e., .683 to '.584, is approximately the^ratio of the corresponding grade- 
point avei^age variances, i.e., .380 to .331. Correlations bet\i;een true 
scores from left to right in Figura 1 are .889, .93^, .897, .942, and .900. * 
These true correlations aj^roximate thofe pr2yioui3ly cstimatad. ^ 

ill. Discussion 



JBteskog (1970a) found that a simplex model provided a fair f it^ to 
Humphreys* (1950) dfita* ^n eight semesters grades in electrical engineering 
(N = 91). Combined with the present results it may be concluded that a «^ 
simplex model provides a good fit to University of Illinois data. Whether 
this woiJld be true fo'r other institutions or for commingled grades from dif-. 
ferent schools remains to be demonstrated. The results support Humplireys* 
conclusion that the reliabilities across semesters were equal. 
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I". Incorporating Nonindependent Me^asAirement Errors 



This section resxa ted from efforts to deal. with correlated 

errors of measurement. Tfliile it is written as if the problem 

<$ - 

were correlated ratings, the same problemi arises whfen the same 

0 

or similar tests are administered over time. The models in tjiis 
-section can be incorporated ±xit(> the simplex framex^ork by 
specifying the dimensions over^time to haye a almplex structure 
as shown in section C. * • * 
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It is frequently the case that e:jq)ert is asked to rate the sam^ 
objects along two oir mo>'e dimensions. In these circumstances it is 
difficult for a judge to npt'let ratings on a dimension be influenced by 
k^fowledge of fatings on other (Jlmensions . ' This kind of contamination means 
* that the errors of measurement for one dimension may be correlated with 
the errors on other dimensions, i.e, the IntrajudRe measurement errors 
are correlated. Under 'these conditions, the covar lance between ratings 
of different dimensions by the same judge is not equal*" to the covariance 
between the underlying true scores as would normally be assumed in 
pla^ssical test theory. The usual correction for attenuabion formula for 
obtai^iing an estimate of the correlation between the underlying true scores 
on the dimensions would be inapplicable since xiricorrelated errors are 
asstimed in'^that formula. Presented herein is a procedure .for analyzing 
data with correlated intrajudge and lincorrelated inter judge measurement 
errors. In addltioa(to testing the fit of the model to the^data, this 
procedure estimates correlations between the true scores on the dimensions, 

- • , ■ ' ' , ■ • 

the reliabilities for each judge on each dimension,- and the correlations 
^between intrajudge errors. ^ 
I. Problgm^ Formulation . 

Let S be the rating of the i— judge (1-= 1 .... N) on the j^S- 

Ij ^ . ' : 

dimension ( j = 1 . . . M) . • ^ 
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^ SupptJ'se that 



where b^^ =1 regression weight of X^^ on > 

■ T, = true score for^the j — dimensi^t^ 

' * ' ^ • ' th ^ ■ : 
and e^j = error of measurement for the i judge on the 

,th dimension. / 

The e . . are assumed to have a mean of zero and to be uncorrelated 

with the T/' . For convenience, • the variance of T, is set equal to unity. 

At this st^ge the model Is similar to the traditional test theory model 

'except that b is not assumed to be the same for all judges as would be 
ij . ^ 

true in the case of parallel fneasure's, and no assumption has been made 
about. the' correlation of the errors of measurement. ^ 

The allowance for correlated intrajudge measurement errors jnea'ns 
that for a given i, errors (e...^ for different values of j are correlated. ' 



It is assumed, however that the interjudge errors are unco^rre la&d which 
means that all e,. for different values of i are uncorrelated. 

0 A factor analytic model is appropriate for analyzing these data, 

however, because certain eirrors are correlated it is computationally 

'■#'■■• • • ' 

convenient to treat the e -as factors along with the'T.. The dispersion 

matrix (Z) of the X., has the form: - ' 
; * . ' ' 

' Z = A $ A' - ' . (2) 

where $ = variance-covarlance matrix among the factors (T^ and e_^'^} , 

' and A = matrix of factor coefficients of the X^^ on the specified 

factors • . . 



52 



ERIC 



^ • ■ .t^ ^ ' ' ■ ^ ' 

It Is njecessary to h^ve at least three independent judges In order 

^ • • ^ ' . •• fl • : . • 

for the correlations among the and error ocovarlances to be uniquely 

estimated, I.e., for the model to b'^e Identified (Fisher, 1966). With • 

only two judges the elements of A •'and * cannot be uniquely estimated and* ,, 
, ^ *• ♦ . ^ 

no test af the model fit Is possible without additional assumptions. 

Le^ S be the dbperved correlation matrix among X . ♦ Fit of this 

modeX [l»e., equation (2)] will be judged by the deviation of the best 

... ■ • . ^ I , ; . 

fit estimate of E from S. For large san^les it is also possible to 

test this fit. ^ . a - 



II, The Three Judge, Two DjLmeftsion Model 



"S 

' . Because- of identification requirementsf 'it is expected that the three 
judge, .two dimension model will be the basic building block for structures 
of this type. c 
The basic equations are: . ^ 
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•Qxe vector of factors is [T^^, T^, e^^, ^j^g' ^^V' '^-xi' "32 
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and the A and $ matrices in equation (2) have the following .form: 
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where the b. . and the * elements are parameters to be estimated. 

in r ^ . 

=W ■ ■ ; 

The, specifications for $ indicate that the "basic dimensions T^^ and T 

are standardized by assigning a variance of unity to the corresponding 

■ 

diagonal elements, which mfeans that the covariance of these factors [$o] 
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la a carrelation/ The six error variances [i.e., *i, $3, $5, $7 and $9] 
ai34 three intrajudge error covariances [i.e., *2 > *5and ^el "® ^® 
estiniated. , 

In order to explore the identif lability of parameters it is useful to 
perform the matrix multiplication Indicated in equation (2) and examirxe 
the entries in E for the above specifications. 

The diagonal entries of S are variances and ate given by: 

The V(e . . ) .are equal to particular diagonal elements in * ; for example, 

V(e- ) = and V(e-.) = *3 • 
11 . 
. The off-diagonal elements of E that correspond to a single dimension 

(j = constant and i'' + k) are; • . 

Given three judges it follows that 

„ C(X, .X_.) C(X, .X,.) 
= I3 -22 13 3j 

C(X2.X3j) 

C(X,,X,.)C(X^^X3.) 
. ^2j - C(X^.X3j) 



_ C(X,,X3.) C(X,.X3.) 



<5 



n ■ ■* 

Since the bf can be expressed in terms of the eletEents in I it follox^s 
that these factor coefficient^ (reliabilities) are identifiable. This 
in turn means that the V(e^^) are idTentified from equation (3) • 

The off-diagonal elements corresponding to different judges* and 
different dimensions (i Ic, 1, 2) are:' a. 

where CitjT^) ° I/q . 

Since all' b and b are identified as shovm in (A) , it follows thkt 

X J. t 

t 

C(Tj^T2) is identified. 

Finally, the off-diagonal elements corresponding to a single judge ^ 

A 

(1 = constant) and different dimensions are: ^ 

where the C(e^j^e^2) ^^"^^ *2 » * 5 ^nd $3 fo'^ ^ ^^^^^ ^' 2 and 3, 
respectively. SS^nce b^^^, h^^, and CiT^T^^ were shown above to be 
identifiable,^ it follows from these equations that the'c (e^j^e^2) 
identified. 

This model has 6 x 7 t 2 ^= 21 unique elements in E and 16 model 
parameters (6 in A, 10 in *) which means that there are 21 - 16 = 5 
degrees of overidentification. Overidentification is necessary to test 
the fit of th§ model to the data. 

As specified above the error variance" for a given rater and dimension 
is obtained from one of. the dia^gal entries of the $ matrix- This for- 
mulation is convenient for investigating the question of identification 
as x^as done above. For purposes of estimation and interpretation, however, 
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and ^ 



J 



The (J) *s are obtained from the (Jj's in the usual manner chat a correlation 
is obtained from covariance and variance terms, e.g. 




It is the latter ^specification of A and^(f) that is used in the emi^irical 

example presented belox-j. 

III. Empirical Example 
If 

Joreslcog*s (1970) general method for analysis of covariance structures 

. y 

and its associated computer program (Joreslcdg, Gruvaeus, & van Thillo., 
1976) were used tfor estima4:ion. An earlier program for restricted maximum 
likelihood factor analysis (Joreskog & Gruvaeus, 1967) or a more recent 

II 

program for estimating a linear structural equation >system (Joreskog & 



so be used* Details oi 



van Thillo, 1972) could also be used. Details of the program ^re given 

J ' ' ' ^ • 

in the manual. • ^ 

: - . 

^ To Illustrate the computations, data provided by Dr. Donald Roclc • 
were used In which three judges -rated thirty-four military positions on 
two dimensions [T^ = Dealing with people and T'^ = Responslbl lity/ Autonomy 1 . 
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Intrajudge errorcT of EeaGurement i^ere probably correlated. Tlie model 

' 4 

for those data ic that given in section II, above. Th^ observed cor- 
relation matri:: S is given in Table 1. The model was set up to yield 
correlations betoeen the. errors I Maxiraura likelihood estimates of model 
parameters in A and 0 are ^gjven in Table 2. . 



Table 1. The Observed Correlation Matrix 





hi 


^12 


^21 


hi 


hi 


^11 


1.0000 










^12 


.5851 


1.0000 








^21 ■ 


.2462 

0 


.1218 


1.0000 






^22 


.4110, 


.5360 


.2709. 


1.0000 




hi ■ 


.3823 


.2946 


..2033 


.0694 


1.0000 


hi 


.2.816 


.6114 


.1675 A 

0' 


.5049 ' 


■ .3314 



X 



32 



4 



According to the ^del,/ reliability is defined as the Square of the 
• • if ' 

corresponding regression coefficient, ^.g., the fir^t dimension rater 
'reliabilities were (.766)^- .387, .110, and .236, respectively. The 
correlation betoeen underlying true dimensions is estimated as .603 and * 
the correlati9ps between intrajyidge errors of measurement are .513, .177, 
and .265, reppectively. The correlation (.513) beti-jeen- errors for the 
first judge approaches the true correlation (.605) betwee^ dimensions ^ 
indicating the necessity for methods which, allow for such contingencies. 



Table 2. Ilaxlmum Likelihood Parameter Estimates 
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A.crucj^al part of the outfiut is the estimated value of 2 and the 



corresponding discrepancies from the observed matrix S, If these are so 

- ■* * • * * • 

large as to indiea{:e that the data do notjit ^he model, then th^ above 
parameter estimates would have lifetle meaning. "^^Ilhe resi'dials^of S - E 
are given in Tafile 3. 
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' Table 3. 


Residuals 
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Considerlng the relatively small sampje size (only 34 persons rated 
by the ^three judges on the tv;o dimensions) and the number of restrictions 
on the model, we Intei^p^Wt these results as a reasonably good f;Lt of; the 
model to the datgr. The goodness of fi^t test yielded a chi squaij^e of 4.54 
with 5 degrees of freedom. The probability of getting a chi-squared 

value larger than that actually obtained, given that the hypothesized 

'■ 

model is true, is P = .475. Since this chi-square tedt assumes a large 
sample, the small sampje irtdicates that these results be interpreted v;ith 
caution. In any event the chi-squared results do not indicate that the 
modeJb\should be rejected because of poor fit to the data. 
V. Discussion^ 



The model analyzed above v/as devi&ed for the rating situation in 
x^hich correlated measurement errors are likely. It may, however, provide 

an appropriate simulation in a varic^ty of other situations, e.g.: 

<^ . , * • 

1. In tl]^e multitrait-multimethod procedure (Campbell & Fiske, 1959) 

. the errors of measurement bet-wRen tvjo* different trait measures using the 
same meLliod may be correlated because of method variance. Method variance 
would be equivalent to correlated intrajudge errors when method factors 

.are uncorrelated vjith trait factors. 

■ ** • ■ 

^ 2. In the use of the same test at two different times, the errors 

oi measurement over time may be correlated because of practice and recall 

effects. At least three different' measures of the underlying construct 

would be necessary for analysis. s|ich a model would be appropriate for 

the study o§ change over time by appropriate formulation of the factors 
ft 

(W^rts, Joreskog, & Linn, 1972). 
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The raodpl does not explicitly state the cfeuses of the- correlation 
between the intrajudge errors, hovievet it is assumed that the causes for 
each set} of errors are uncorrelated with the causesN^^or the other, sets 

and of the dimensions being measured. A good fit of the model H:o the 

p ■ , 

. o ■ ■ 

data implies that these assqmptions are consistent with the results. ° 
Insight may be gained into the meaning of a good fit VJith the data 
by examining the equations of the three judge, ti/o dimension model for 
the variables Xj^^, ^^21^ ^31 ^12* section II it was demonstrated 
that the three measures' of T , i.e., , X^^^ and X Identify the three 



^ regression weights .of ratings on T- : 



mi 



2 ^^hiV 

. 11 C(X2,X3^)^ 



2 ^^hihi^ ^^hihi^ , 
^21 = c(x^^X3p 



, 2 _ ^^^lAi^ ^^%Si^ 



^ '^1 - ccx^^x^p . . 

In factor analytic language, given three m^astiires of a single factor with 
uncorrelated residuals, the three factor loadings may be uniquely estimated, 
Hoxiyever with only three measures of a factor, no test of the assumption of 
single factoredness is possible be'cause a perfect fit x^ith the data is al- 
ways achieved (although 'communalities greater than one may be re^u^red) . . 
Thus, the model haff no degrees of overidentif ication. It is of interest 
therefore to examine the relationship among ^2±* ^31 ^^12 ^^^^^^ yield: 



o2 
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2 ^<V21> ^^hlhl^ • 

21 " ; C(X^2V 

2 ^ ^(^2^31> ^<^2Al) 
31 - , C(X,2X2,) • ^ 



0- 



2 2 

Comparison of the pairs of equations for h^^ and b^^^ indicates that 
'for the purposes of identifying and estimating these loadings, X^^ is 
functionally equivalent to X^^^^. ' In other vjords, even though not 
a measur^e'of T , it nevertheless allows a test of the hypothesis of . 
single-factoredness of T^, The basic reason for this is that e^^^ 
uncorrelated vjith T^, e^^, and e^^^^even though it i& correlated x^ith 
^11' '^^^ finding 6f a good fit to the data is therefore consistent^ 
with the assumption that the observed variables are in fact measures 
of the specified dimension. A poor f if might be due to the falsity 
of this assumption, ho^li^er one or jnore of the o£her model assumptions 
may be erroneous. 

The variety of models involving correlated errors is too great to 
be detailed herein. For most of these the three judge, two dimension 
model is likely to be the basic unit. Within the cpnstraints set by 
the computer program, the available data should, however, be analyzed 
by a single model. For example, three judge, three dimension data 
could be computed using the three Judge, two dimension modfel for each 
of the three different pairs of dimensions. The result would be that 
for each reliability two estimates would be obtained which might differ 
considerably. A simultaneously estimated thi^ee judge, three dimension 
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model would yield a single best fit estimate for all the data. Providing 
a good fit. is obtained, a parameter derived from the three judge, three 
dimension model should have greater generalizability because it implies 
that the tx^o estimates from the .correspottdihg three judge, tx^o dimension 
models are consistent. ^ 



64 



f 



Campbell, D. T. , & Flske, D. W. Convergent and discriminant .'Validation' 
hy validation- by the multitralt-tnultlmethod matrix. Psychological 
* Bulletin , 1959, 56, 81--105. 

Fisher, F. M. .The identification problem in econometrics . New York: 
McGraw-Hill, 1966. . 

Joreskog, K. G.', & Gruvaeus, G\ T. RMLFA - A computer program for 

restricted maximum likelihood factor analysis. RB-67-21 . Princeton 
N. J.: Educat;ional Testing" Service, 1967. 

Joreskog., K. G. A general' method for analysis of covariance s'tructures. 
Biometrika , 1970, 52, 239-251. 

Jareskog, K. G. , Gruvaeus, G. T.., & van ThilliD, M. ACOVS - A general' 
computer program for analyses of covariance structures. ' RB~70-i5 « 
' Princeton, N.. J.:. Educational Testing Service, 1970. 

II ^ ' - 

Joreskog, K. G. , & van Thillo, M. LISREL - A general computer program 

for estimating a linear structural equation system involving multiple 

indicators of unmeasured variables. RB~72~56 . Princeton, N. J.: 

Educatioiial Testing Service, *i972. 

Werts, C% E., Joreskog^ K. G. , & Linn, R. .L. A multi trait-multimethod 
model for studying growth. Educational and Psychological Measure- 
ment, 1972, 32, 655-678. ^ 



Conclusions • 

/ - • 

The most Important finding of this study/ is that a simplex 
model which allows for measurement error, tits a variety of longi- 
tiidinal academic d^ta quite well. As detailed in section D, 
this allows for attenuation corrections when only one measure is. 
available at each time. More importantly, ' the results suggest • 
that the commonly used split-half or parallel form procedures 
for estimating reliability may typically yield overestimates of 

o 

reliability due to "method" variance i.e. , n on in dependent 
measurement errors resulting from the use of closely similar item 
types. The simplex model appears less subject to this problem 
because both item formats sccid content change, over time. It has 
been demonstrated that accurate corrections foV attenuation are 
essential to a study of the determinants of academic growth. 

These initial remits ha^e encouraged us to incorporate 
the simplex model' into larger s):ructural models* with favorable 
results. Furthermore, the probleift of combining simplexes for 
different miasures was found feasij5l^\. These resultis will be 
forthcoming in the literature as soon as the analyses are 
completed. 



